Fix integration test failing on first Dependabot PR run#692
Conversation
When custom Lambda layers are defined, Serverless Framework calls CloudFormation.describeStacks in compareWithLastLayer() to check whether the layer has already been uploaded. Its error handler only silences "does not exist" responses; any other error – including the "security token invalid" response AWS returns when credentials are absent – is re-thrown, failing sls package. Add a lightweight test-only Serverless plugin (offline.js) that intercepts CloudFormation.describeStacks during packaging and returns a synthetic "does not exist" error. This makes compareWithLastLayer treat every run as a fresh stack, which is the correct behaviour for snapshot tests where we never actually deploy. The snapshot output is unchanged because S3 key timestamps are already normalised. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #692 +/- ##
=======================================
Coverage 77.69% 77.69%
=======================================
Files 12 12
Lines 1112 1112
Branches 350 350
=======================================
Hits 864 864
Misses 118 118
Partials 130 130 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ava-silver
left a comment
There was a problem hiding this comment.
The fix is correct and well-explained — the root cause analysis is solid and the offline.js shim does what it needs to do. One thing worth doing alongside this (or in a follow-up) is a naming pass, because the mislabeling is what makes this shim look suspicious in the first place.
These tests never deploy anything — they run sls package, generate a CloudFormation template, and diff it against a checked-in snapshot. That's a snapshot test, not an integration test. Calling them "integration tests" implies real AWS connectivity is expected, which is exactly why the credential failure looked like a bug rather than an obvious non-issue.
The naming is scattered across several places that would all need updating together:
integration_tests/directory →snapshot_tests/(orpackaging_tests/)integration_tests/run_integration_tests.sh→snapshot_tests/run_snapshot_tests.shintegration_tests/serverless-extension.yml/serverless-forwarder.yml→ move accordingly- The CI workflow file and its job name — both currently called
integration-testsin the Actions runs - The
integrationTesting/testingModeplugin option insrc/env.tsis a separate concept (it bypasses forwarder ARN validation), so that one is fine to leave alone
For context on why there's no cleaner upstream fix: --noDeploy was removed in Serverless Framework v2.36.0 and sls package is the recommended replacement — which is exactly what these tests already use. The compareWithLastLayer() credential issue has been an open bug since serverless/serverless#8187 (2020), and there's an open feature request for a --artifacts-only flag at serverless/serverless#12969 (Dec 2024) with no movement. So the shim is the right call for now — it just deserves a comment pointing at those issues so future maintainers know this isn't accidental and know when it can be removed.
Co-authored-by: Ava Silver <ava.silver@datadoghq.com>
The entire PR (including the summary below) was generated by Claude Code.
Problem
The
integration-testsCI job fails on the first run of every Dependabot PR with:This happens during
sls packageforserverless-extension.yml, but not forserverless-forwarder.yml. Dependabot PRs run without valid AWS credentials, which causessls packageto fail when it tries to call AWS.Root cause
When
serverless.ymldefines a top-levellayers:section, Serverless Framework v3 callsCloudFormation.describeStacksfor each layer incompareWithLastLayer()(inlib/plugins/aws/package/compile/layers.js). This is an optimisation that checks whether a layer has already been uploaded to avoid re-uploading unchanged ones. The error handler in that function only silences errors whose message contains"does not exist"— any other AWS error is re-thrown:The forwarder test is immune because our plugin wraps all CloudWatch calls in a broad
catchthat returns[], silently absorbing credential errors. The extension test has no such safety net.Fix
Add a lightweight test-only Serverless plugin (
integration_tests/offline.js) that hooks intobefore:package:compileLayersand interceptsCloudFormation.describeStackscalls, immediately rejecting them with a synthetic"does not exist"error. This makescompareWithLastLayertreat every snapshot test run as a fresh stack, which is the correct behaviour for snapshot tests where we never actually deploy.serverless-extension.ymlis updated to load this plugin.How I tested it
Confirmed the root cause by running
sls packagelocally with placeholder invalid credentials against both configs:Verified the fix by running the same command after adding
offline.js:Verified snapshot is unchanged by applying all the normalisation steps from
run_integration_tests.shto the generated template and diffing againstcorrect_extension_snapshot.json— no differences.Cross-referenced with PR history: PR #666 (a previous Dependabot PR) had the identical failure on its first run and was fixed only by pushing a
run cicommit to trigger a fresh CI run, confirming this is a consistent first-run issue rather than a one-off flake.